Systems and methods for creating an interactive 3d visualization of indexed media

ABSTRACT

A 3D graphical user interface includes a two-dimensional ground-plane layout representing the relationship between one or more leaf elements of a tree data structure The interface further includes at least one building-like structure, each of the at least one building-like structures corresponding to a respective one of the one or more leaf elements. Each of the at least one building-like structure provides a summary of media associated with the respective one of the more leaf elements corresponding to that at least one building-like structure.

BACKGROUND OF THE INVENTION

1. Field of Invention

This invention relates to systems and methods for creating aninteractive 3D visualization that indexes media and summarizes theindexed media.

2. Description of Related Art

Based on increases in conventional computing speed and resourcemanagement, three-dimensional media indices are just recently becomingviable means for indexing a large amount of related media. However, thecomplexity of the index and information about the indexed media conveyedby the three-dimensional indices is rather limited.

For instance, SGI FSN is a file system navigator with a 3D landscapevisualization. Directories are placed in a hierarchical tree layout inthe landscape. Each directory in SGI FSN is represented by a pedestal.The size of the files in the directory is proportional to the pedestalheight and the file age is shown by color. Files are placed on top ofthe directory pedestals. Each file is represented by a box.

U.S. patent application [Attorney Docket 114890] describes a method forcreating and updating interactive 3D visualizations of media indiceswherein separate indices are located above and below a ground plane. Theindices are organized in a table structure defined by two axes andprovide links to the media that is indexed. Various visualcharacteristics of the indices reflect characteristics of the media thatis indexed as well as various user defined or group defined informationrelating to the indexed media.

Perspective Wall is a 3D visualization that indexes files. It utilizes a3D wall with three sections and displays card icons for the files.3D/Rooms is a 3D index that has a 3D multi-level building floor plan inwhich users can select a research group and retrieve photos of membersand their offices.

Finally, There are many visualizations of the World Wide Web using 3Dlandscapes. Most of these such as WWW3D, WebPath, and VR-VIBE useabstract geometric objects to represent documents. The Web Forager,utilizes a space that contains 3D bookcases and 3D books, that whenopened, display the indexed Web pages.

SUMMARY OF THE INVENTION

Each of the above-described systems for indexing media within athree-dimensional space shares a similar shortcoming. Particularly, auser is only able to discern very general, if any, qualities of theindexed media from the visualization itself. For example, according toU.S. patent application [Attorney Docket 114890] a user can determinehow the media is classified within two axes or determine whether or notthe media indexed is one of or a combination of graphics, text, video,etc. However, none of the above-described systems summarize the indexedmedia within the summary itself, such that a user can view a summary ofthe indexed media simply by viewing the 3D visualization of the indexedmedia.

Therefore, various exemplary embodiments of the systems, methods, andgraphical user interfaces according to this invention provide aninteractive 3D visualization wherein the surfaces of 3D objectsrepresenting indexed media provide visual summaries of the indexedmedia.

Various exemplary embodiments of the systems and methods according tothis invention provide a 3D graphical user interface including atwo-dimensional ground-plane layout representing the relationshipbetween one or more leaf elements of a tree data structure; and at leastone building-like structure, each of the at least one building-likestructures corresponding to a respective one of the one or more leafelements, wherein each of the at least one building-like structureprovides a summary of media associated with the respective one of themore leaf elements corresponding to that at least one building-likestructure.

Various exemplary embodiments of the systems and methods according tothis invention provide a layout for a 3D visualization includinginputting a tree data structure, each leaf of the tree associated withone or more forms of media; determining a plurality of candidatelayouts; comparing the plurality of candidate layouts; selecting asolution layout; and constructing a 3D visualization based on thelayout, wherein the representation of each leaf within the 3Dvisualization provides a summary of the media contained by that leafwithin the input tree data structure.

BRIEF DESCRIPTION OF THE DRAWINGS

Various exemplary embodiments of systems and methods according to thisinvention will be described in detail, with reference to the followingfigures, wherein:

FIGS. 1 and 2 show an exemplary embodiment of a 3D visualization thatutilizes substantially square and similarly sized building-likestructure plots;

FIGS. 3-6 show a number of variations of the 3D visualization shown inFIGS. 1 and 2;

FIGS. 7 and 8 show an exemplary embodiment of a 3D visualizationaccording to this invention that utilizes differently sizedbuilding-like structure plots;

FIG. 9 shows a graphical user interface for use with the 3Dvisualizations of FIGS. 1-6;

FIG. 10 shows a graphical user interface for use with the 3Dvisualization of FIGS. 7 and 8;

FIGS. 11 and 12 are a flowchart outlining an exemplary embodiment of agrid layout method for the type of 3D visualizations shown in FIGS. 1-6.

FIG. 13 shows an example of leaf nodes layouts and branch nodes layouts;

FIG. 14 shows the layouts of FIG. 13 in which the leaf node layouts arecombined into a single layout;

FIG. 15 shows an example of the layouts of FIG. 14 combined into largerlayouts;

FIG. 16 shows an exemplary embodiment of a functional block diagram of a3D visualization system;

FIGS. 17-21 are a flowchart outlining an embodiment of a method fordetermining layout for the type of 3D visualizations shown in FIGS. 7and 8; and

FIG. 22 shows an exemplary embodiment of a functional block diagram of a3D visualization system.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIGS. 1 and 2 show an exemplary embodiment of a 3D visualization 100according to this invention that utilizes substantially square andsimilarly sized building-like structure plots.

As shown in FIGS. 1 and 2, the 3D visualization utilizes a city metaphorin which each amount of media, indexed by a leaf in a tree datastructure, is represented by a building-like structure 120. According tothis exemplary embodiment, the indexed media is video. However, anyother media such as text, graphics, charts, etc. may be used. Theindexed media is summarized on the roof 122 and walls 124, 126, and 128(rear not shown) of the buildings such that a user navigating the 3Dvisualization has an idea of the contents of each leaf without having toaccess that leaf. In various exemplary embodiments, the roof 122 andsides 124, 126, 128 (rear not shown) of the buildings display the samesummary. Alternatively, each of the roof 122 and sides 124, 126, 128(rear not shown) have different summaries. In various exemplaryembodiments, a video is summarized by arranging key frames of the videoin a predefined rectangular area (here defined by the height and widthof the sides and roof of the buildings). FXPAL-IP-98-018 and U.S. patentapplication Ser. No. 10/392,822 describe methods for automatic videosummary using key frames and are incorporated herein by reference intheir entirety. Key frame video summaries work particularly well withinthe context of a city metaphor because each of the summary's framesresemble windows on the side of the building-like structures.

As shown in FIGS. 1 and 2, and consistent with the city metaphor, eachbuilding-like structure 120 exists on a lot 110 wherein a curb width 140is the distance from the building-like structure to a road. In variousexemplary embodiments, the curb width is inversely proportional to anumerical value associated with each indexed portion of the media. Forinstance, the numerical value may represent the length of the video oran importance of the video determined by the user, or a predefined groupof users. As such, the indexed portions of media that have the highernumerical values will have a smaller curb width and the building-likestructure will be larger.

It should be appreciated that, in other exemplary embodiments of the 3Dvisualization according to this invention, the numerical value may bedirectly proportional to the building-like structure's foot print.However, in practice this may result in building-like structures thatare too thin to effectively summarize their respectively indexed media,or building-like structures that are too wide and obstruct a user's viewof other buildings.

As shown in FIGS. 1 and 2, and consistent with a city metaphor, a roadseparates each lot 110. In various exemplary embodiments, the road width(e.g., 130, 132) represents the distance between the respective indexedmedia portions within the underlying tree data structure. For example,lots 110 that are separated by roads with a road width 130 are closer toone another within the tree data structure (e.g., leaves) than lots 110that are separated by roads with road width 132 (e.g., braches).

Furthermore, in various exemplary embodiments, the height 250 of eachbuilding-like structure is proportional to a numerical value associatedwith each indexed portion of the media. For instance, the numericalvalue may represent the length of the video or an importance of thevideo determined by the user, or a predefined group of users.

In various exemplary embodiments, as shown in FIG. 1, text may beincluded in the visualization to provide labels 150 for various groupsof indexed media. In various exemplary embodiments, these labels floatover the building-like structures to simulate clouds, consistent with acity metaphor.

FIGS. 3-6 show a number of exemplary variations 300, 400, 500, and 600of the visualization 100 shown in FIGS. 1 and 2. According to thevariations 300, 400, 500, and 600, each leaf represented by abuilding-like structure is summarized by up to four types of media(single video frames 310, key frame summaries 410, slide images 510 andtext 610). In each exemplary variation 300, 400, 500, and 600, adifferent type of media is displayed on the roof. In various exemplaryembodiments of the invention, the type of media that most effectivelysummarizes the indexed media portion is displayed on the roof because,typically, the roof is most visible from viewpoints in which all or manyof the building-like structures are visible.

FIGS. 7 and 8 show an exemplary embodiment of a 3D visualizationaccording to this invention that utilizes differently sizedbuilding-like structure plots wherein the size of each building-likestructure's plot is proportional to some numerical quality of the indexmedia represented by that building-like structure.

Similar to the previously described embodiments of FIGS. 1-6, thebuilding-like structures have curb widths and building heights that areproportional to some numerical qualities of the indexed media portion(leaf) represented by the building-like structures. Again, similar tothe previously described embodiments, FIG. 7 shows road widths 732 thatrepresent the distance between the respective indexed media portionswithin the tree structure.

As shown in FIGS. 7 and 8, according to this exemplary embodiment theplots 710 of each building-like structure for each leaf are arrangedinto larger plots 740, 760 wherein each larger plot 740, 760 includesall of the plots representing its subnodes according to the underlyingtree data structure (e.g., leaf plot 710 is contained in larger plot 740which is contained in larger plot 760). Text may be included in thevisualization to provide labels 150 for various groups of indexed media.In various exemplary embodiments, these labels 150 float over thebuilding-like structures to simulate clouds, consistent with a citymetaphor.

In order to facilitate the use of the above-described 3D visualizationaccording to various exemplary embodiments of the invention as agraphical user interface, various exemplary embodiments provide a numberof functions that facilitate the effective viewing of and interactionwith the various portions of the 3D visualizations. Various exemplaryembodiments, enable a user of the visualization to change the viewingangles, viewing distance, and view point. For example, FIG. 1 shows thevisualization of the first embodiment, in which the view is relativelydistant and tilted to primarily display the fronts 124 and tops 122 ofthe building-like structures Alternatively, FIG. 2 shows thevisualization of the first embodiment in which the view is relativelyclose and tilted to primarily display the fronts 124 and tops 122 of thebuilding-like structures. A user may change the viewing angle by, forexample grabbing the visualization and rotating it with a pointingdevice, touch screen, or virtual reality interaction devices such aglove, by using keyboard commands, or by any other known or laterdeveloped input device.

Furthermore, various exemplary embodiments allow a user to access theindexed media portions by clicking on the various summaries on thebuilding-like structures. For example, clicking on a single video frame310 or a frame within a key frame summary 410 will play the indexedvideo beginning from that frame. Similarly, clicking on a slide 510 willaccess a larger view of the indexed slide or an indexed group of slides.Clicking on a text summary 610 will access the document or documentsthat the text summary 610 represents.

Alternatively, as shown in FIGS. 9 and 10, various exemplary embodimentsof the invention provide graphical user interfaces 900, 1000 including anumber of slides 920 and 922 and buttons 924, 926, 928, 930, 932, 934,936, and 938 that enable a user to navigate and interact with thevisualization. FIG. 9 shows one exemplary embodiment of a graphical userinterface for use with the 3D visualization of FIGS. 1-6. FIG. 10 showsone exemplary embodiment of a graphical user interface for use with the3D visualization of FIGS. 7 and 8. In FIGS. 9 and 10, buttons 936, 934,932, 938 move the viewpoint forward, backward, left, and right,respectively. Buttons 928 and 930 zoom the viewpoint in and out. Buttons924 and 926 move the viewpoint up and down (in altitude). Slider 922tilts the view. Slider 920 is a controller does several operationstogether; it changes altitude and tilts the view, such that the point atcenter of the screen is kept fixed. As such, slider 922 enables the userto see the rooftop when high up, to see the front of the building whennear the ground, and to maintain the building in the center of thescreen in view.

FIGS. 11 and 12 show a flowchart outlining an exemplary embodiment of agrid layout method for the above-described type of 3D visualization thatutilizes substantially square and similarly sized building-likestructure plots according to this invention. As discussed above, thedata to be visualized is organized as tree of elements where each branchin the tree is represented by a relatively square rectangle and all ofthe elements within this rectangle are part of that branch of the tree.This exemplary embodiment of a layout method balances the desire tominimize blank space in the layout with the need for a relatively squarelayout to match the shape of the majority of output devices, e.g., acomputer monitor, a printer, or the like. A single line layout wouldproduce no blank space but would not be an efficient use of outputdevice real estate.

As shown in FIGS. 11 and 12, operation of the method begins in ‘stepS1000 and continues to step S1005 where each leaf of the tree isassigned a similar substantially rectangular area. Then, in step S1010,a first/next node whose subnodes are all leaves is selected as thecurrent node. Next, in step S1015, the leaves of the current node arearranged into a larger rectangle such that the ratio of the height towidth and the ratio of the width to height is less than a predeterminedaspect ratio and that larger rectangle is assigned to the current node.The predetermined aspect ratio is used to insure that the resultingrectangle is relatively square. This prevents the creation of rectanglesthat are long and thin. Operation of the method continues to step S1020.

In step S 1020, a determination is made whether the current node is theroot node of the tree. If the current node is the root node, all of theleaves of the tree have been arranged within a rectangle and operationjumps to step 51085 where the operation of the method ends. However, ifthe current node is not the root node, operation continues to step S1025.

In step 51025, a determination is made whether all of the nodes whosesubnodes are all leaves have been selected as the current node. If allof the nodes whose subnodes are all leaves have not been selected as thecurrent node, operation returns to step S1010. However, if all of thenodes whose subnodes are all leaves have been selected as the currentnode, operation continues to step S1030.

In step S1030, a first/next node that has at least one larger rectangleassociated with one of its subnodes and a rectangle associated with eachsubnode is selected as a current node. Then, in step S1035, adetermination is made whether a current node has more than one leafrectangle and more than one larger rectangle associated with itssubnodes. For example, as shown in FIG. 13, there are five leaf nodes1300 and three branch nodes (larger rectangles) 1305, 1310, 1315. If thecurrent node does not have more than one leaf rectangle and more thanone larger rectangle associated with its subnodes, operation continuesto step S1040. However, if the current node has more than one leafrectangle and more than one larger rectangle associated with itssubnodes, operation jumps to step S1055.

In step S1040, the various rectangles, both leaf rectangles and largerrectangles, associated with the subnodes of the current node arearranged into larger rectangles such that the ratio of the height towidth and the ratio of the width to height is less than thepredetermined aspect ratio. Then, in step S 1045, a determination ismade whether a successful layout was found such that each of therectangles fit within a new larger rectangle with a minimum of emptyspace. If a successful layout is not found, operation to continues tostep 51050, where the aspect of ratio is increased. Then, operationreturns to step S 1040. However, if a successful layout is found,operation jumps to step S 1080.

In step S1055, the various leaf rectangles associated with the subnodesof the current node are arranged into a larger rectangle such that theratio of the height to width and width to height is less than thepredetermined aspect ratio. For example, as shown in FIG. 14, the leafnodes of FIG. 13 are combined into a single larger rectangle 1400.[EXPAND ON ADVANTAGE OF THIS STEP] In various exemplary embodiments, itis advantageous to group the leaf nodes into a larger rectangle becausethey represent media files in the same directory in the directory treeand therefore should remain close together, rather than scatteredthroughout the layout. Next, in step S1060, the larger rectangles,including the larger rectangle composed of the various leaf rectanglesassociated with the subnodes of the current node, are arranged intovarious larger rectangles such that the ratio of the height to width andwidth to height is less than the predetermined aspect ratio.

For example, as shown in FIG. 15, the various rectangles 1305, 1310,1315, 1400 are combined into a number of larger rectangles 1500, 1510,1520. For the purpose of this example, the aspect ratio is 1.5 (theheight cannot be more than 1.5 times the width and vice versa) andrectangles 1500, 1510 and 1520 show the three larger rectangles thatwould be considered (4×6, 5×5, and 6×4).

Then, in step 51065, a determination is made whether a successful layoutwas found such that each of the rectangles fit within the new largerrectangle with a minimum of empty space. As shown in FIG. 15, rectangle1500 encloses each of the smaller rectangles 1305, 1310, 1315, 1400 withonly three units of empty space. Rectangle 1510 encloses each of thesmaller rectangles 1305, 1310, 1315, 1400 with four units of emptyspace. Rectangle 1520 cannot enclose each of the smaller rectangles1305, 1310, 1315, 1400. It should be appreciated that, in variousexemplary embodiments, when the rectangles have the same amount of emptyspace, preference is given to the rectangle with the smallestcircumference. Returning to FIGS. 11 and 12, if a successful layout wasnot found, operation continues to step S 1070, where the aspect ratio isincreased. Then, operation returns to step S1055. However, if asuccessful layout was found, operation continues to step S1080.

In step 51080, the successful layout is assigned to the current node.Then, in step S 1085, a determination is made whether the current nodeis the root node. If the current node is the root node, operationcontinues to step S 1090 where the method ends. However, if the currentnode is not the root node, operation returns to step S 1030 where thenext node that has at least one larger rectangle associated with one ofits subnodes and a rectangle associated with each subnode is selected asthe current node. Operation of the flowchart will continue until, instep S 1085, the current node is the root node.

FIG. 16 is an exemplary embodiment of a functional block diagram of anexemplary embodiment of a 3D visualization system 1600 that is usable tocreate 3D visualizations according to the invention. As shown in FIG.16, the 3D visualization system 1600 includes an input/output interface1620, a controller 1625, a memory 1630, a node identifying circuit,routine, or application 1640, a rectangle assigning circuit, routine, orapplication 1645, a rectangle arranging circuit, routine, or application1650, an aspect ratio adjusting circuit, routine, or application 1655, alayout testing comparing circuit, routine, or application 1660, and amedia associating circuit, routine, or application 1665, eachappropriately interconnected by one or more data/control busses and/orapplication programming interfaces 1670, or the like.

In this exemplary embodiment, the input/output interface 1620 isconnected to one or more input devices 1605 over one or more links 1601.The input device(s) 1605 can be one or more of a keyboard, a mouse, atrack ball, a track pad, a touch screen, a virtual reality glove, or anyother known or later-developed device for inputting data and/or controlsignals to the 3D visualization system 1600. Furthermore, in thisexemplary embodiment, the input/output interface 1620 is connected toone or more output devices 1610 over one or more links 1602. The outputdevice(s) 1610 can be one or more of a computer monitor, cathode raytube, liquid crystal display, image projector, electrophoretic display,a virtual reality display device, or any other known or later-developeddevice for visually displaying 3D visualization data output from the 3Dvisualization system 1600.

In this exemplary embodiment, the input/output interface 1620 isconnected to a data source 1700 over a link 1701. The data source 1700can be a locally or remotely located laptop or personal computer, apersonal digital assistant, a tablet computer, a device that storesand/or transmits electronic data, such as a client or a server of awired or wireless network, such as for example, an intranet, anextranet, a local area network, a wide area network, a storage areanetwork, the Internet (especially the World Wide Web), and the like. Ingeneral, the data source 1700 can be any known or later-developed sourcethat is capable of providing a media data organized in a tree to theinput/output interface 1620.

Each of the various links 1601, 1602, and 1701 can be any known orlater-developed device or system for connecting the input device(s)1605, the output device(s) 1601, and/or the data source 1700,respectively, to the input/output interface 1620. In particular, thelinks 1601, 1602, and 1701 can each be implemented as one or more of adirect cable connection, a connection over a wide area network, a localarea network or a storage area network, a connection over an intranet, aconnection over an extranet, a connection over the Internet, aconnection over any other distributed processing network or system,and/or an infrared, radio-frequency or other wireless connection.

As shown in FIG. 16, the memory 1630 contains a number of differentmemory portions, including a layouts portion 1631, a tree structureportion 1632, a leaf data portion 1633, a solution layout portion 1634,and an aspect ratio portion 1235. The layouts portion 1631 of the memory1630 stores various layouts. The tree structure portion 1632 of thememory 1630 stores the tree structure of the data that will berepresented in the 3D visualization. The leaf data portion 1633 of thememory 1630 stores the media represented by each leaf in the treestructure. The solution layout portion 1634 stores the solution layoutthat will be used in the 3D visualization. The aspect ratio portion 1235stores the aspect ratio used to determine the various layouts.

The memory 1630 shown in FIG. 16 can be implemented using anyappropriate combination of alterable, volatile or non-volatile memory ornon-alterable, or fixed, memory. The alterable memory, whether volatileor non-volatile, can be implemented using any one or more of static ordynamic RAM, a floppy disk and disk drive, a writeable orre-re-writeable optical disk and disk drive, a hard drive, flash memoryor the like. Similarly, the non-alterable or fixed memory can beimplemented using any one or more of ROM, PROM, EPROM, EEPROM, anoptical ROM disk, such as CD-ROM or DVD-ROM disk, and disk drive or thelike.

The node identifying circuit, routine, or application 1640 accesses thetree structure data and determines the characteristics of a node. Therectangle assigning circuit, routine, or application 1645 assignsrectangular layouts to leaf nodes. The rectangle arranging circuit,routine, or application 1650 arranges rectangular layouts into variouslarger rectangular layouts, based on the aspect ratio. The aspect ratioadjusting circuit, routine, or application 1655 adjusts the aspectratio. The layout testing circuit, routine, or application 1660 testsvarious layouts to based on the amount of empty space. Finally, themedia associating circuit, routine, or application 1665 associates theleaf data with the solution layout to create the 3D visualization.

In operation, the exemplary embodiment of the 3D visualization system1600 shown in FIG. 16 operates in the following manner. Under control ofthe controller 1625, the data necessary for creating the 3Dvisualization is input form the data source 1700 across the link 1701via the input/output interface 1620 and stored in the tree structure andleaf data portions 1632, 1633 of the memory 1630. Next, under control ofthe controller 1625, the node identifying circuit, routine, orapplication 1640 accesses the tree structure data in the tree structureportion 1632 of the memory 1630 and identifies each of the leaf nodes.Then, under control of the controller 1625, the rectangle assigningcircuit, routine, or application 1645, accesses the identified leafnodes and assigns each leaf node a similar substantially rectangulararea. The rectangles for each leaf node, under control of the controller1625, are associated with their respective leaf node and stored in thelayouts portion 1631 of the memory 1630.

Under control of the controller 1625, the node identifying circuit,routine, or application 1640 accesses the tree structure data stored inthe tree structure portion 1632 and the layout data in the layout dataportion 1631 and identifies each node whose subnodes are all leaves.Then, under control of the controller 1625, the rectangle arrangingcircuit, routine, or application 1650 accesses the identified subnodesand arranges the rectangles assigned to the leaf nodes of eachidentified node into a larger rectangle. The rectangles for theidentified nodes, under control of the controller 1625, are associatedwith their respective identified nodes and stored in the layouts portion1631 of the memory 1630.

Under control of the controller 1625, the node identifying circuit,routine, or application 1650 accesses the tree structure data stored inthe tree structure portion 1632 and the layout data in the layout dataportion 1631 and identifies each node that has at least one rectangleassociated with each subnode, at least one of the rectangles beinglarger than a leaf rectangle. Of these identified nodes, under controlof the controller 1625, the node identifying circuit, routine, orapplication 1640, further identifies a first subgroup of the identifiednodes that have more than one leaf rectangle associated with itssubnodes, and more than one rectangle larger than a leaf rectangleassociate with its subnodes.

Under control of the controller 1625, the rectangle arranging circuit,routine, or application 150 accesses the first subgroup of theidentified nodes, and for each node, arranges the various leafrectangles associated with the subnodes of that node into a largerrectangle with the smallest circumference. Then, under control of thecontroller 1625, the rectangle arranging circuit, routine, orapplication 1650, for each node of the of the first subgroup ofidentified nodes, arranges the larger rectangles of the subnodes of thatnode into rectangles whose height/width or width/height is less than theaspect ratio stored in the aspect ratio portion 1635 of the memory 1630.When, for a particular node, the larger rectangles of the subnodes ofthat node will not fit into any rectangle according to the aspect ratio,the aspect ratio adjusting circuit, routine, or application 1655accesses the aspect ratio in the aspect ratio portion 1635 and increasesthe aspect ratio for that node until the larger rectangles of thesubnodes of that node will fit into at least one rectangle with thesmallest aspect ratio.

Then, under control of the controller 1625, the layout testing circuit,routine, or application 1660 accesses the various rectangles for eachnode of the first subgroup of identified nodes and selects the rectanglefor each node which has the least amount of empty space. If more thanone rectangle for a node of the first subgroup of identified nodes hasthe same amount of empty space, the layout testing circuit, routine, orapplication 1660 will select the rectangle with the smallestcircumference (i.e., closest to square). Under control of the controller1625, the selected rectangle for each node of the first subgroup ofidentified nodes is associated with its respective node and stored inthe layouts portion 1631 of the memory 1630.

Once a rectangle has been stored in the layouts portion 1631 for eachnode of the first subgroup of identified nodes, the rectangle arrangingcircuit, routine, or application 1650 accesses the remaining nodes ofthe identified nodes as the second subgroup of identified nodes (i.e.,having a rectangle associated with each subnode and a rectangle largerthan a leaf rectangle associated with at least one subnode, but notincluded in the first subgroup of identified nodes). Then, under controlof the controller 1625, the rectangle arranging circuit, routine, orapplication 1650, for each node of the of the second subgroup ofidentified nodes, arranges the larger rectangles of the subnodes of thatnode into a rectangles whose height/width or width/height is less thanthe aspect ratio stored in the aspect ratio portion 1635 of the memory1630. When, for a particular node, the larger rectangles of the subnodesof that node will not fit into any rectangle according to the aspectratio, the aspect ratio adjusting circuit, routine, or application 1655accesses the aspect ratio in the aspect ratio portion 1635 and increasesthe aspect ratio for that node until the rectangles of the subnodes ofthat node will fit into at least one rectangle.

Then, under control of the controller 1625, the layout testing circuit,routine, or application 1660 accesses the various rectangles for eachnode of the second subgroup of identified nodes and selects therectangle for each node which has the least amount of empty space. Ifmore than one rectangle for a node of the second subgroup of identifiednodes has the same amount of empty space, the layout testing circuit,routine, or application 1660 selects the rectangle with the smallestcircumference (i.e., closest to square). Under control of the controller1625, the selected rectangle for each node of the second subgroup ofidentified nodes is associated with its respective node and stored inthe layouts portion 1631 of the memory 1630.

Once a rectangle has been selected and stored in the layouts portion1631 for each node of the identified nodes (i.e., both the first andsecond subgroups), under control of the controller 1625, the nodeidentifying circuit, routine, or application 1640 again accesses thetree structure data stored in the tree structure portion 1632 and thelayout data in the layout data portion 1631 and identifies each nodethat now has at least one rectangle associated with each subnode, atleast one of the rectangles being larger than a leaf rectangle. Thesenodes become the new set of identified nodes, and the data flowcontinues as above. However, when at any time during the operation ofthe 3D visualization system 1600, under control of the controller 1625,the node identifying circuit, routine, or application accesses the treestructure data stored in the tree structure portion 1632 and the layoutdata in the layout data portion 1631 and determines that there is only asingle layout associated with a single node, the solution layout hasbeen determined (all layouts have been combined until the root node isassociated with the only rectangle).

Then, under control of the controller 1625, the single layout for theroot node is stored in the solution layout portion 1634 of the memory1630. Next, under control of the controller 1625, the media associatingcircuit, routine, or application 1665 accesses the solution layout inthe solution layout portion 1634 and the media data in the leaf dataportion 1633, and associates the media data for each leaf with thatleaf's location according to the solution layout. At this point the 3Dvisualization has been created and, under control of the controller1625, is ready to be output to the output device(s) 1610 across link1602 via the input output interface.

It should be appreciated that, depending on cost or other designconstraints, one or more of the above-described elements of the 3Dvisualization system 1200 may be combined into a single element ordivided into multiple elements where appropriate.

FIGS. 17-21 are flowcharts outlining an exemplary embodiment of a methodfor determining layout for the above-described 3D visualization,exemplified in FIGS. 7 and 8, wherein each leaf node is assigned adifferent size parcel that reflects an attribute of that leaf node.

As shown in FIG. 17, operation of the method begins in step S2000 andcontinues to step S2005 where the root node is selected as the currentnode.

Then, operation continues to step S2010, where a determination is madewhether the current node is a leaf node. If the current node is a leafnode, operation continues to step S2015, where a solution layout for theleaf node is constructed with a given width and height. Then, operationcontinues to step 52170. However, if the current node is not a leafnode, operation continues to step S2020 where the first or next subnodeof the current node is selected as the current node. Operation continuesto step S2025.

In step S2025, a determination is made whether the current node is aleaf node. If the current node is a leaf node, operation continues tostep S2030 where a layout is constructed for the leaf node with a givenwidth and height. Otherwise, operation jumps to step S2040. Next, instep S2035, the constructed layout is assigned to a new set of layoutsassociated with the parent node of the current node. Then, operationjumps to step S2065.

In step S2040, a determination is made whether all the subnodes of thecurrent node have been selected as a current node. If all subnodes ofthe current node have not been selected as a current node, operationreturns to step S2020. If all subnodes of the current node have beenselected as a current node, operation continues to step S2045. In stepS2045, a determination is made whether more than one set of layouts isassociated with the current node. If there is more than one set oflayouts associated with the current node, operation jumps to step S2070.If there is not more than one set of layouts associated with the currentnode, operation continues to step S2050.

In step S2050, a determination is made whether the current node is theroot node. If the current node is the root node, operation continues tostep S2055. If the current node is not the root node, operation jumps tostep S2060.

In step S2055, the “smallest” layout is selected from the set of layoutsas the solution. In various exemplary embodiments, the “smallest” layoutis determined to be the layout with the smallest circumference. As such,the “smallest” layout may not have the smallest overall area, but willhave a small area and be the closest to square. Operation then continuesto step S2170.

In step S2060, the set of layouts that is associated with the currentnode is associated with the parent node of the current node. Then,operation continues to step S2065, where the parent node is selected asthe current node. Next, operation returns to step S2040, where it isonce again determined whether all of the subnodes the current node havebeen selected as a current node.

In step S2070, an empty new set of layouts is created for the currentnode. Next, in step S2075, a set of layouts containing the smallestlayout is selected from the sets of layouts associated with the currentnode as set 1. Then, in step S2080, a set of layouts, other than set 1,containing the smallest layout is selected from the sets of layoutsassociated with the current node as set 2. Next, step S2085, a first ornext layout is selected from set 1 as the current layout of set 1. Then,in step S2090, a first or next layout is selected from set 2 as thecurrent layout of set 2. Operation continues to step S2095.

In step S2095 the current layout of set 1 and the current layout of set2 are combined horizontally adjacent to one another, separated by agiven amount of space, to create the current combined layout. Next, instep S2100, determination is made whether the new set of layouts for thecurrent node is empty. If the new set of layouts for the current node isempty, operation continues to step 52105 where the current combinedlayout is added to the new set of layouts for the current node. Then,operation jumps to step S2145. However, if the new set of layouts forthe current node is not empty, operation continues to step S2110.

In step S2110, a first or next layout from the new set of layouts forthe current node is selected as the current new layout. Then, operationcontinues to step 52115, where it is determined whether the currentcombined layout is “larger” than the current new layout. According tovarious exemplary embodiments of this method, “larger” means that onelayout has both a larger width and a larger height than another layout.If the current combined layout is larger than the current new layout,operation continues to step 52120 where the current combined layout isdiscarded. Then, operation jumps to step S2145. However, if the currentcombined layout is not larger than the current new layout, operationcontinues to step S2125.

In step S2125, it is determined whether the current new layout is“larger” than the current combined layout. If the current new layout islarger than the current combined layout, operation continues to step52130 where the current new layout is discarded. Operation thencontinues to step S2135. However, if the current new layout is notlarger than the current combined layout, operation jumps to step S2135.It should be appreciated that it is possible for the current combinedlayout to not be larger than the current new layout and the current newlayout not be larger than the current combined layout since in order tobe larger both the height and the width of one of the layouts must berespectively larger than the height and the width of the other layout.

In step S2135, a determination is made whether all of the layouts in thenew set of layouts for the current node have been compared with thecurrent combined layout. If not, operation returns to step 52110.However, if all of the layouts of the new set of layouts for the currentnode have been compared with the current combined layout, operationcontinues to step S2140 where the current combined layout is added tothe new set of layouts for the current node. Then, operation continuesto step S2145.

In step S2145, it is determined whether the current layout of set 1 andthe current layout of set 2 have been combined both horizontally andvertically. If not, operation continues to step S2150, where the currentlayout of set 1 and the current layout of set 2 are combined verticallyadjacent to one another, separated by a given amount of space, to createthe current combined layout. Operation then returns to step 52110.However, if the current layout of set 1 and the current layout of set 2have been combined both horizontally and vertically, operation continuesto step S2155.

In step S2155, it is determined whether all of the layouts from set 2have been combined with the current layout from set 1. If not, operationreturns to step S2090 where the next layout from set 2 is selected asthe current layout of set 2. If all of the layouts from set 2 have beencombined with the current layout from set 1, operation continues to stepS2160. In step 52160, it is determined whether all the layouts from set1 have been selected as the current layout for set 1. If not, operationreturns to step S2085 where the next layout from set 1 is selected asthe current layout of set 1. However, if all of the layouts from set 1have been selected as the current layout for set 1, operation continuesto step S2165.

In step S2165, set 1 and set 2 are discarded, and therefore, are nolonger associated with the current node. Then, operation returns to stepS2045, where a determination is made whether there are more than on oneset of layouts associated with the current node, and the operation ofthe flowchart continues as above.

In steps S2015 and S2055, once a solution layout has been determined,operation continues to step S2170. In step S2170, the solution layout isselected as the current layout. Next, in step S2175, a starting x and yare associated with the current layout. Then, in step 52180, it isdetermined whether the current layout is associated with a tree. If thecurrent layout is associated with a tree, operation continues to stepS2185, where the x, y, width, and height of the current layout areassociated with the tree. Operation continues to step S2190. However, ifthe current layout is not associated with a tree, operation jumpsdirectly from step S2180 to step S2190.

In step S2190, it is determined whether the current layout is composedhorizontally, (i.e., was the current layout created in step S2095). Ifthe current layout is composed horizontally, operation continues to stepS2195. If the current layout is not composed horizontally, operationjumps to step S2205.

In step 52195, the x and y of the current layout are associated with theleft component layout, the left component layout being the layout of theleft side of the horizontally combined current layout. Next, in stepS2200, the left component layout is selected as the current layout.Then, operation returns to step S2180.

In step S2205, it is determined whether the current layout is composedvertically (i.e., was created during step S2150). If the current layoutis composed vertically, operation continues to step S2210. If thecurrent layout is not composed vertically, operation jumps to stepS2220. In step 52210, the x and y of the current layout are associatedwith the lower component layout, the lower component layout being thelayout on the bottom of the vertically combined current layout. Next, instep 52215, the lower component layout is selected as the currentlayout. Then, operation returns to step S2180.

In step S2220, a determination is made whether the current layout is thesolution layout. If the current layout is the solution layout, everycomponent layout has been assigned a location and operation jumps tostep S2270, where the method ends. However, if the current layout is notthe solution layout, operation continues to step S2225.

In step S2225, the enclosing layout is selected as the current layout.Then, in step S2230, it is determined whether the current layout iscomposed horizontally. If the current layout is not composedhorizontally, operation jumps to step S2250. However, if the currentlayout is composed horizontally, operation continues to step S2235 whereit is determined whether the right layout has been assigned a positionvalue (i.e., whether or not an x and y is associated with the rightlayout). If so, operation returns to step S2220. However, if the rightlayout has not been assigned a position value, operation continues tostep S2240. In step S2240, the width of the left layout and a givenamount of space is added to the x associated with the current layout.Then, this new value and the y of the current layout are associated withthe right layout. Next, operation continues to step S2245 where theright layout is selected as the current layout. Then, operation returnsto step S2180.

In step S2250, a determination is made whether the current layout iscomposed vertically. If not, operation returns to step S2220. However,if the current layout is composed vertically, operation continues tostep S2255 where it is determined whether the upper layout has beenassigned a position value. If the upper layout has been assigned aposition value, operation returns to step S2220. If the upper layout hasnot been assigned a position value, operation continues to step S2260where the height of the lower layout and a given amount of space isadded to the y associated with the current layout. Then, the x of thecurrent layout and the new value for y is associated with the upperlayout. Next, operation continues to step S2265. In step S2265, theupper layout is selected as the current layout and operation returns tostep S2180. The operation of the flowchart will continue to move thevarious steps until, in step S2220 the current layout is the solutionlayout. At such time, each component layout of the solution layout hasbeen assigned a position value and operation continues to step S2270where operation of the method ends.

FIG. 22 is exemplary embodiment of a functional block diagram of anotherexemplary embodiment of a 3D visualization system 2200 that is usable tocreate 3D visualizations according to the invention. As shown in FIG.22, the 3D visualization system 2200 includes an input/output interface2220, a controller 2225, a memory 2230, a node identifying determiningcircuit, routine, or application 2240, a set associating circuit,routine, or application 2245, a set creating circuit, routine, orapplication 2250, a layout size comparing circuit, routine, orapplication 2255, a layout constructing circuit, routine, or application2260, a horizontal layout combining circuit, routine, or application2265, a vertical layout combining circuit, routine, or application 2270,and a media associating circuit, routine, or application 2275, eachappropriately interconnected by one or more data/control busses and/orapplication programming interfaces 2280, or the like.

In this exemplary embodiment, the input/output interface 2220 isconnected to one or more input devices 2205 over one or more links 2201.The input device(s) 2205 can be one or more of a keyboard, a mouse, atrack ball, a track pad, a touch screen, a virtual reality glove, or anyother known or later-developed device for inputting data and/or controlsignals to the 3D visualization system 2200. Furthermore, in thisexemplary embodiment, the input/output interface 2220 is connected toone or more output devices 2210 over one or more links 2202. The outputdevice(s) 2210 can be one or more of a computer monitor, cathode raytube, liquid crystal display, image projector, electrophoretic display,a virtual reality display device, or any other known or later-developeddevice for visually displaying 3D visualization data output from the 3Dvisualization system 2200.

In this exemplary embodiment, the input/output interface 2220 isconnected to a data source 2300 over a link 2301. The data source 2300can be a locally or remotely located laptop or personal computer, apersonal digital assistant, a tablet computer, a device that storesand/or transmits electronic data, such as a client or a server of awired or wireless network, such as for example, an intranet, anextranet, a local area network, a wide area network, a storage areanetwork, the Internet (especially the World Wide Web), and the like. Ingeneral, the data source 2000 can be any known or later-developed sourcethat is capable of providing a media data organized in a tree to theinput/output interface 2220.

Each of the various links 2201, 2202, and 2301 can be any known orlater-developed device or system for connecting the input device(s)2205, the output device(s) 2201, and/or the data source 2300,respectively, to the input/output interface 2220. In particular, thelinks 2201, 2202, and 2001 can each be implemented as one or more of adirect cable connection, a connection over a wide area network, a localarea network or a storage area network, a connection over an intranet, aconnection over an extranet, a connection over the Internet, aconnection over any other distributed processing network or system,and/or an infrared, radio-frequency or other wireless connection.

As shown in FIG. 22, the memory 2230 contains a number of differentmemory portions, including a layouts portion 2231, a tree structureportion 2232, a leaf data portion 2233, and a solution layout portion2234. The layouts portion 2231 of the memory 2230 stores variouslayouts. The tree structure portion 2232 of the memory 2230 stores thetree structure of the data that will be represented in the 3Dvisualization. The leaf data portion 2233 of the memory 2230 stores themedia represented by each leaf in the tree structure. The solutionlayout portion 2234 stores the solution layout that will be used in the3D visualization.

The memory 2230 shown in FIG. 22 can be implemented using anyappropriate combination of alterable, volatile or non-volatile memory ornon-alterable, or fixed, memory. The alterable memory, whether volatileor non-volatile, can be implemented using any one or more of static ordynamic RAM, a floppy disk and disk drive, a writeable orre-re-writeable optical disk and disk drive, a hard drive, flash memoryor the like. Similarly, the non-alterable or fixed memory can beimplemented using any one or more of ROM, PROM, EPROM, EEPROM, anoptical ROM disk, such as CD-ROM or DVD-ROM disk, and disk drive or thelike.

The node identifying circuit, routine, or application 2240 accesses thetree structure data and determines the characteristics of a node. Therectangle assigning circuit, routine, or application 2245 assignsrectangles to leaf nodes. The set associating circuit, routine, orapplication 2245 associates various layouts with a set of layouts. Theset creating circuit, routine, or application 2250 creates empty setswith which various layouts may be associated. The layout size comparingcircuit, routine, or application 2255 compares the size various layouts.The layout constructing circuit, routine, or application 2260 createslayouts for leaf nodes. The horizontal layout combining circuit,routine, or application 2265 combines layouts horizontally into a largerlayout. The vertical layout combining circuit, routine, or application2270 combines layouts vertically into a larger layout. Finally, themedia associating circuit, routine, or application 2265 associates theleaf data with the solution layout to create the 3D visualization.

In operation, the exemplary embodiment of the 3D visualization system2200 shown in FIG. 16 operates in the following manner. Under control ofthe controller 2225, the data necessary for creating the 3Dvisualization is input from the data source 2000 across the link 2001via the input/output interface 2220 and stored in the tree structure andleaf data portions 2232, 2233 of the memory 2230. Next, under control ofthe controller 2225, the node identifying circuit, routine, orapplication 2240 accesses the tree data stored in the tree structureportion and identifies the leaf nodes. Then, under control of thecontroller 2225, the layout constructing circuit, routine, orapplication 2260 accesses the identified leaf nodes and constructs alayout with a given width and height for each leaf node. Under controlof the controller 2225, the set creating circuit, routine, orapplication 2250 creates an empty set of layouts for each of theidentified leaf nodes. Then, under control of the controller 2225, thelayouts for each leaf node are stored within the new set created forthat leaf node's layout node in the layouts portion 2231 of the memory2230 and associated with the parent node of the leaf node for which theset was created.

After a layout for each node has been created, under control of thecontroller 2225, the node identifying circuit, routine, or application2240 accesses the tree structure data stored in the tree structureportion and identifies each node that only has one set associated withit. Then, under control of the controller 2225, the set associatingcircuit, routine, or application 2245 associates that set with the firstparent node in the identified node's progeny that has more than onesubnodes. Next, under control of the controller 2225, the nodeidentifying circuit, routine, or application 2240 identifies all of thenodes that have a set of layouts associated with each of that node'ssubnodes.

Under control of the controller 2225, the set creating circuit, routine,or application 2250 creates a new empty set of layouts for each of theidentified nodes. Then, under control of the controller 2225, for eachidentified node, the horizontal layout combining circuit, routine, orapplication 2265 selects a layout from the set of layouts containing thesmallest layout (set 1). Then, for each identified node, under controlof the controller 2225, the horizontal layout combining circuit,routine, or application 2265 identifies the set of layouts other thanset 1 with the smallest layout (set 2). Under control of the controller2225, the horizontal layout combining circuit, routine, or application2265 combines each layout from set 1 with every layout from set 2 in ahorizontally adjacent fashion. The first of these combined layouts isstored in the newly created empty set of layouts for each of identifiednodes. Then, under control of the controller 2225, the remainingcombined layouts for each identified node are accessed by the layoutsize comparing circuit, routine, or application 2255.

The layout size comparing circuit, routine, or application 2255, undercontrol of the controller 2225, compares each combined layout with eachof the layouts in the newly created set. If, under control of thecontroller 2225, the layout size comparing circuit, routine, orapplication 2255 determines that a combined layout is “larger” than alayout in the new set (i.e., both the height and width of the combinedlayout is larger than the layout in the new set), the combined layout isdiscarded. If, under control of the controller 2225, the layout sizecomparing circuit, routine, or application 2255 determines that a layoutin the new set is “larger” than a combined layout, the new layout isdiscarded and the combined layout is stored in the new set.

Next, under control of the controller 2225, the vertical layoutcombining circuit, routine, or application combines each of the layoutsfrom set 1 with every layout from set 2. Again, under control of thecontroller 2225, if the layout size comparing circuit, routine, orapplication 2255 determines that a combined layout is “larger” than alayout in the new set, the combined layout is discarded. If, undercontrol of the controller 2225, the layout size comparing circuit,routine, or application 2255 determines that a layout in the new set is“larger” than a combined layout, the new layout is discarded and thecombined layout is stored in the new set.

Once, under control of the controller 2225, each of the layouts from set1 has been combined vertically and horizontally with all of the layoutsfrom set 2, sets 1 and 2 are discarded. Next, the node identifyingcircuit, routine, or application 2240, under control of the controller2225, accesses the tree structure data stored in the tree structureportion 2232 and the layout data stored in the layout portion 2231 andidentifies each node that only has one set associated with it. Then,under control of the controller 2225, the set associating circuit,routine, or application 2245 associates that set with the first parentnode in the identified node's progeny that has more than one subnodes.Next, under control of the controller 2225, the node identifyingcircuit, routine, or application 2240 identifies all of the nodes thathave had a set of layouts associated with each of that node's subnodes.

Under control of the controller 2225, the set creating circuit, routine,or application 2250 again creates a new empty set of layouts for each ofthe identified nodes. In this manner, for each node, the sets of layoutsassociated with that node will be consolidated until one set remains andthat set will be associated with the nodes parent node. The data flowrepeats until, under control of the controller 2225, the nodeidentifying circuit, routine, or application 2240 determines that theroot node has only one set of layouts associated with it and all of thesubnodes of the parent node no longer have any sets. At this point,under control of the controller 2225, the layout size comparing circuit,routine, or application 2255 accesses the set of layouts associated withthe root node and selects the smallest layout from the set (i.e., thelayout with the smallest circumference). Then, under control of thecontroller 2225, the smallest layout is stored in the solution layoutportion 2234, of the memory 2230. Next, under control of the controller2225, the media associating circuit, routine, or application accessesthe solution layout in the solution layout portion 2234 and the mediadata in the leaf data portion 2233, and associates the media data foreach leaf with that leafs location according to the solution layout. Atthis point, the 3D visualization has been created and, under control ofthe controller 2225, is ready to be output to the output device(s) 2210across link 2202 via the input output interface 2220.

It should be appreciated that, depending on cost or other designconstraints, one or more of the above-described elements of the 3Dvisualization system 1200 may be combined into a single element ordivided into multiple elements where appropriate.

According to the above-described exemplary embodiments of the systems,methods, and graphical user interfaces according to this invention, a 3Dvisualization of media indexed within a tree data structure is providedwherein 3D objects representing the indexed media display visualsummaries of the indexed media. These summaries allow a user to visuallyevaluate the indexed media, based on its content, without having toaccess the media itself. The ability to evaluate the media based on itscontent is particularly advantageous when a large amount of media isindexed and it would be impractical to access each leaf of the tree datastructure in order to evaluate the media.

While the invention has been described in conjunction with exemplaryembodiments, these embodiments should be viewed as illustrative, notlimiting. Various modifications, substitutes, or the like are possiblewithin the spirit and scope of the invention.

1.-33. (canceled)
 34. A 3D graphical user interface comprising: atwo-dimensional ground-plane layout representing the relationshipbetween one or more leaf elements of a tree data structure; and at leastone building-like structure, each of the at least one building-likestructures corresponding to a respective one of the one or more leafelements, wherein each of the at least one building-like structureprovides a summary of media associated with the respective one of themore leaf elements corresponding to that at least one building-likestructure.
 35. The 3D graphical user interface of claim 34, wherein eachof the one or more leaf elements is represented by a similarly sizedsubstantially square plot within the two-dimensional ground plane. 36.The 3D graphical user interface of claim 34, wherein each of the one ormore leaf elements is represented by a differently sized substantiallysquare plot within the two-dimensional ground plane.
 37. The 3Dgraphical user interface of claim 36, wherein dimensions of eachdifferently sized substantially square plot reflect one or morecharacteristics of the media associated with the leaf elementcorresponding to that differently sized square plot.
 38. The 3Dgraphical user interface of claim 34, wherein each of the at least onebuilding-like structure provides the summary of media associated withthe respective one of the more leaf elements corresponding to that atleast one building-like structure by displaying a portion of that mediaon at least one of the sides or on the top of the building-likestructure.
 39. The 3D graphical user interface of claim 38, wherein thesummary of the media comprises at least one frame of a video.
 40. The 3Dgraphical user interface of claim 38, wherein the summary of the mediacomprises a slide.
 41. The 3D graphical user interface of claim 36,wherein the summary of the media comprises text.
 42. The 3D graphicaluser interface of claim 38, further comprising text floating over thebuilding-like structures.